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BSTRflCT 

As part of an effort to provide more economical 
ethods for job-skill evaluation in the Army Enlisted Personnel 
anagement System, a study investigated the use of television as a 
eans of presenting test items for a sample of tests from the job 
ieia of tank crewman.. The television scene provided the job setting 
or each item and then posed a question that required a real-time 
esponse from the examinee. All items were job-connected and . 
herefore represented simulated skill items, ■ The examinee responded 
irectly to the face of the television screen by means of an 
lectronic stylus or gun reticle. The test was compared with a 
aper--and-pencil test which covered the same items and a hands-on 
erformance test which covered many of the same items. The results 
ndicated at television testing is very acceptable to soldiers and 
easible* ae validity of the prototype test could not be determined 
recisely ^ecnuse the criterion hands-on test was not usable; most 
xaminees made a perfect score on the hands--on test. (Author/GDC) 



Reproductions supplied by EDRS are the best that can be made * 

from the original document. 



Technical Paper 303 



us DEPARTMENTOF HEALTH 
EDUCATION A WELFARE 
NATIONAL INSTITUTE OF 
EDUCATtON 

^n,S DOCUMENT HAS tU" E N WhflUO- 

Tur F'F WSON Of^ OHGANUAI ION OPiCiN. 
A T I N C. I T P O I N T S O f V I T O W O P i N t ON S 
ST A TED DO NOT NKCFiSSAWiL V W f. f k t - 
SE N ■ f IC I Al NATIONAL 1 NM i T U.T F C)» 

^ Due A T ic:n f'OS" r ion ow roi k y. 



AD 



TELEVISION AS STIMULUS INPUT IN 
SYNTHETIC PERFORMANCE TESTING 



John T. Cockrell 



SCOPE OF INTEREST NOTICE 

ThR ERIC F^ciliTv has auigned 
thii riurMrnp.-it for pfore^ 



T/n 



In our ludgcrrKjnt, this riocumant 
IS .Jlso of interMt to the ctoaf ing- 
h'-susei noted to the right. Indax- 
r.hould rnfleci thoir spocia! 



ARI FIELD UNIT AT FORT KNOX, KENTUCKY 




00 
O 

o 



ERIC 



U. S. Army 

Research Institute for the Beiiaviora! and Socia! Sciences 

August 1978 

\ 

y 

Apf) rf) veti for pul5l:c i(?l(?nst?; d ; s r r i h u I lo n unhmiTC;] 





U S. ARMY RESEa\RCH INSTITUTE 
FOR THE BEHAVIORAL AND SOCiAL S ENCES 




A Field Operating Agency under the Jurisdiction ot the 
Deputy Chief of Staff for Personnel 



PIS ■■ •■ ! GL^TtON Pnm^srv div!r 'Lvjtion c-f rhis .ef.-M hcs i^een : ;;v ARI Ploase address correspondencf: 
'■.onccrrnng ciistnbuTion of report wo, i J . ^; . Army Rtr-ear'h In- : ;jte for the Beh av.oral ftnd Social Sciences, 
ATTN Pf.i''\ r, f>0f)1 Fi'>t'nh,iv\^?'- Av .;:< , Ai . Rnr Jm,-? Vir/ji' .^fia^ 



fi^j^-J * - -Q-L'3 P_Qj5J TJ CN This rfiporr n^av be dostro^ec .'^ •! is no longer Meeced. Pleas*? do nru return ti !o 
me U "5 Army Reseorch !rt8?iTuTe for The aahoviorftl and L">o:;i5l Sciences. 






Wll.MAM MAI iSi:i< 
C'f)lt)r}cl . l.JS Ai n}\ ' 
CoiiiniatuicT 




■NQl^E: Tne f.'.dmgj m this report arc; noTto be construed as ar\ official DepH-'tmenf fvP Army poJ'tiori, 
unless so designeied by other futhorizec? documents 



SECUPllTY CLASSIFICATION OF THIS PAGE (Whetn l^^tn h:ntrrrfl)_ 



REPORT DOCUMENTATION- PAGE 



1 RtPORT KUMHKR 



b. OOVT At L:t.f>SION NO 



READ INSTRUC TIONS 
HKKORK COMPLI^TING KOF^M 



4 r I r L. I- I tnut Suhtltio) 



7. AUTHOR^*) 



•k 1 - 1 I 



*9 PLRfORMINO ORGANIZATION N AM f: AND A D U R f; 5 S 

' '", S . A]-iir/ ••■'(•:■• \\ J ^'ii 1 n : 1. i t u t r.^ for t'lf 



i "iia V 1 o r. 1 1 ail' 



)i ' i . I 1 ^u - i ( M L( 



'ji)('I t : i : ',1 1< '1 raw* • ! Av»ii\i!', AU-x.iiulrMa^ V i . 



II C:.)N T rOL.L I N' O O F F !( . t NAMtl AND AOORFSS 

■ 'rnm<ir>'. i ( Ti-'A.] / ' ] 



TV^^^^Tro^Jit^"'"^ ft AODRESS^/f (ry//rrmw from - '< ^r. f re ; / « (>//.rr» 



3 ^U Clf'IF:NT'£, rATAUOr, NUM13F-R 



rypr: or f-if POF^r a F>f'f<ioo covERro 



6 H ERf ORMIN G ORG. RC.rORJ NUMlltH 



0. CONTRACT OR GRANT NUMatlRfa* 



H). PROGRAM t^LF.MtNT, PROJEC r, TASK 
ARFA a WORK UNIT NUMBf. RS 



t?; REPORT DATF^ 



AucTiist- 1 976— 



13. NUMBER OF PAGES 



1&. SECURITY CLASS, (nf thin roporO 

UlU'l a:- ■ f i-cd 



Asvi', ntCL A ncATiON^ downgrading 

SCHED> E 



> )Vu'd !"■- r pail 1 i c r 



rl i st' r i but". Loii mi^ 1 imi ta^c^. 



10 r.UPPL tMENT ARY NOTES 



1?. -•< t. Y WORDS Con f ID u?* on rir:-frnc> ^ 

i^Ti bit 1 {jo r- To rmanv^^ : t.a.atiiKj 

S VII t:" 1^ :• I" L c fji. r lA )rmanca^ L'a^'-Liu?, 
TV' 1 i a lOii t '^a i ru; 
Job-:;:, i 1 1 t.ta^t:^iaia 



^/n ^f nrrc.isa/y /irn.^ ifirrUify by ht<u-k nttrr.hnr) 



^a :)rirt:. '^f ar^, Mfforti'tc^ provide inor^- r:^c;or:omlraI inot:hod.s for -job-^^kill 
'>va. baat :r>n \n tlvv :bi 5-'. ta?d Porr,onaol Manaaoment System (bPMS), a- study 
i 1. a-.) r od t/ac- u.^-/ oiT television as means of i.^rovidiaq test item 

: aea->nt.at i.en . Th.- television scene provided the job settin<i fo^ each item 
and then pe^i^^jd a <:ueation that required a rcal-ti^me response from the 
-xamine,:-. All it^-nis were iob-connected and ther^'^fore represented simulated 
:.,k M 1 i'om.s. Vir- examinee responded directlv>eo ^:he face of the television 
-^ere.^' 1 means nf an (^lec c stylus or c^n reticle. The test was 

7^ 



DO rjA>! 1473 EDfTIOS T NOV *5 JS OBSOLETE 



UTiclas'v.^ i f ied 



SECURITY CL ASSrFlCATtON OF THIS PAGE (Wh<m Da/» En(Bred) 



ERLC 



1 



lj!LL:.la:.!; i f j >.Mi 

SECURITY CL ASSIFiCATION OF THIS PAGEr«7i«n Dmt» E^^t»f»d) 



< • ^ u ' V 1 w i t i i ,1 p> ii ' I n r 1 - f ' M' ■ i 1 f i ^ : ; 1 wl i ; '! i < '< v/i i t^i\ t I c • : uin ' i i ' 'id • , ii 1 

. I i 1 IIK 1: ; - I Ml t )f ' I t'nr IIM Mr •( . t , > ; ; t wl 1 i ^ i ( ■ t i \'i ' v , . . i UMMV- * ' t til'' ' I 111* ' i t i 'Hi; • Tl l< ' 

! * 1 t :. M\u i t i t !i, i t ( • ' 1 ' V i i ^ Ml t ( ' : : t i n< I 1 . > \N 1 ■ • •( •< 'J H , il > ! • • n . : -.M ! , I i , . f-:-. 

iM-i r*M.ii]^l.'. Til' vMlitiit-/ tln^ { ;t r • .< > r-i^jM not h- < -f. • t . ^ r nu n- H 

1 ■ I'' ■ i ; - • 1 y 1 -r - Mil.'-.* • t!i- orit"!"i<Mi lMnw::-oii w.i.': ii(»t ur,,ihl.'. 



EKLC 



fJnc lass i ^ iod 



SECURITY CL ASSIFfCATlON OF THIS PA,OE(WhBn Datit Endrtid) 



5 



Tet leal Paper 303 



TELEVISION AS STIMULUS INPUT IN 
SYNTHETIC PERFORMANCE TESTING 



vJohn T. Cockrell 
' and 

Donald M. Kristiansen, Work Unit Leader 



ARI FIELD UNIT AT FORT KNOX, KENTUCKY 



Submitfed as complete and 
iGchnicaiiy accurate, by: 
Donald F, Haggard 
Field Unit'Chief 



Approvec 

E. Ralph Dusek, Director 

INDIVIDUAL TRAINING AND PERFORMANCE 
RESEARCH LABORATORY 



Joseph Zeidner 

TECHNICAL DIRECTOR (DESIGNATE) 



U.S ARMY RESEARCH INSTITUTE FOR THE BEHAVIORAL AfJD SOCIAL SCIENCES 
bOOl Eisenhower Avenue, Aiexandria. Virginia 22333 



Office, Dc[)Ljtv Chief of Staff for Personnel 
Dopiirtment of the Arnny 



August 1978 



Army Project Number 
2Q763731A77^ 



'j! formance-Oriented 
Individual Training 



EKLC 



Apprt>vcd for puWic release; distribution unlimited. 



Af^l f^f>s«Mi(:l) H(»|)orts '.Hiii TecfintCiil P.i()oir> arr uitf»rul(Ml ior SfXJFisors of 
fUSil^ tijsks <in(i ottx.T r<?s(Nir(:li c\n{\ inililary .iq<?fu:M»s. Any trfulni(|s riu\dy for 
irnplfMnnnlation M Iittk* of puhlicntiofi .hc f)r(»S(»nlf.'<l m [\]v laller part ol 
Itu^ BruM. Uf)()n coniplotton of a major ptiasc of ttu^ task, torfT)aI roromoHMi 
(l-itions for officicjl acMion norrTially iUV convryod to ap()ro()riale oiilitary 
afjnncies by l)ricfir)(| or Disposition f orrn. 



t i.Miiii-i r i .1 » ■ r ' n-.- a I 111 i i n ! i vm < ii k i 1 ; ;< ) j ( i i < • r' ; ; t'cM'in.. a iiMjfW i' 

t "^MilV :■:.•:.(/. U 1 M t 1 ■ : : : f 1 ( \ I t < ' for t ll^' t u ^ } M V i t ) I . I 1 , in- i T^l )( M ,1 1 : " r i i ' 31 ' . • ^ ( Af ■ [ f . 

'!'!;■ !v • f t o frnai ; -t' i " 1 1 1 < 'i 1 1 lui i v i 1 1 ua 1 .Skill P- ^ Vt \I t ^| m* - n t a:i< 1 iiva I i m t u t i 
f.r"i^i<M'r i:i /^("ni' 't MI i< ^< I v;it;i i :nj )T"i i v i in f .t"< ' 1 < ■ Viiin » , < ^ J" f i c i <.Mn ' y , ainl « m '(.>ni mip/ 
of nidi.vi^lu.i 1 •■iilist"! liMiniiH] <iM'] < ^ va 1 u 1 1' i on . I'n a rna jra- t ci 

Mn}Mi)\r^' t •.■'inilMt i ntv^'-, r>r > 1 ^ i i •aa. , ttx' Arin\' is i > 1 ♦ ■as^ mi t i aq Mi.- 

l-:r. 1 I: ; f f •< I I't ■ > aMan* > 1 : la a an - mium) t y; ;taMii (I'lPM:'. ) . 'f'h i a ay a. t • mii r < ^aa i - a. 

r < >af T 't ilia iia llic iraliv^Maal traiaip<i ant] ti-;tia,'i a.y : ; I - -ii'.a i ^ in ila - th^'ni 

1 oh i \ > 1 K'V.iw \ . 

I-:va 1 iiat- i<ai o t a- i lul i. 7.1 ilaa I sold it,"; r;-; lor rat"*'t'r pa-( x: ] r< 'a ; ; i ^ )n iai tha 
I'iPMS i.a. baS'.M;! o:\ ^a^ i ta'> r 1 o!^. - ra^ fa rancaa ]j(^rf <n n\af\(a^ ta'».at in(| ol aaLnal ia]> 
:;killa fat. liar than ' fa 1 1< ^ ra 1 .i./;( al knowlodcfa. I'.a ^'Tfort. lo aaii Lava laarc 
ar.r>noiny in tJu: 1 a rqt^ -a;r\a 1 a ta^a.t;in(} rc<iuirtal for t.lua 1-:PMS , a ria-;acna;li 
procjram a^M-kiaq ko (Javalop siinulataHi j lar f orrnanco taast:.^"; k.ar> InaMi ini.- 
Li.ataal. Ona pi-ornia^inq liau' ^> f iau"loavor la tho. vvm2 of varia-nia audio- 
vi.:.iu<-i] nu'dia i.ct i.a.v'vLda i i i< ■ si.rrivulu:^ inuuP uiad jolj a* ; l I, ;i ac; (or tlia 
.ak i- i ] itaanr:. '["ha [)ra'a(Mit ]ju1; 1 i cat. ja-)n rofjortaa tho result a of a sturiy 
iTi v^aat Lqa t: LucT Ih;^ uaa T t^Mc^ v .1 a ion s t. .i. mu 1 vv- iaiputa \u coii ]uuat:i on witdi 
an r> laatron 1.^: r(^:;pondina vr}h i c J - v/h i ah rofral.ro^". ra-al -f-iina d(^^aiaion.a and 
I ■ ai^.iaaiaaa. . 

Thi'- r'a:.a-irr-h w a": .ioTV^' at Id'ia MU I'^l'Md UnLt at'Forf Knox, Kv., iw 
roapon!;a ta^ r^'ou i ar'^- n ta; of Ar;iiy Pro iatd" 2od , 37 ^ 1A770 and tn >^]aM.;iaL 
roqurr^'Rit.aita-; of { )aa '['rairiiaiq and Pr)r' r i no 'yoraraap.d (TPAPOd) , l-'ort Mor^ro", 



1^ ,>^\^iu^>- 



iViainical Director (Designate) 



ERIC 



T=' ■ in • • I i ., 1. 1 1 . t r i- ' V. I li ' i i t y ■ i n ^ i f < ' ^ i m 1 « i 1 i I \/ « > i u ; ' i r i < f t < • I r ■ i ^ ; i on 
. t I !nn n ■ i i p ,m ' • : in n ! n/n i 1 h M i < • w 1' < M'lih uu ■ t ■ ' ; t , ni n 1 t ( > ( 1* ' t. * • i^'ii i nt » i 1' 
n I • 1 1 S : ! ; m n r" ^ ■ n 1 . \ i ■ ' * . i n - 1 - 1 ■> n p. ■ t 1' i > mik ^ , . ' f ■ 1 • ' n t : ; . ' I '1 1 i n i ^ r, ■ ■ i t l ■ 1 1 i : ; 
^n ■ • 1 . h • r ■ .! n' ' '. n I r \- ^ ■ -n n: ■ « - ! ! i' ■ ' i i ^ jh ^ ■ ! f > t li. 1; ; - on I • • i n nnM 
n- ' n. ■ . f ■ . ) , i- ' ■ 1 « -n n i , • ■ . ■ xi 'M: . i , r i 'n ■ m nl > 1 v \\ i 1 i J . n. 1 1 ' ; t i I n t ' ' . 



I' ! >.-. •. Hi : 

A s'/'-fii^'t ii- I " ' r I ' Tinnii' "( ' tt-.nt ns i.nq ( j ' 1 \' i s in"^ii n:: t lu- st i luu I un iri[.Mit 
iioV'M^ir^i-^l nji'l ) > r"» n li, i( -t m i . ^rlio t-nM wnn ; -ons i ^ r^.- i a ijf-- r" rr)T;T;iatu ■( ' 
l'^n{. bu/.\i.u::<^ t lu' it. 'mhs iT>vrr«nl 1; In* <i'Mu<il tcL'^V^:'^ tlw <'xainiiuMn^ W(''r'^ 
r« V {u i. tn^s 1 '\'.f> pt'T'M'rn.m on tin,' job. '^']\t^ tn.-^st: v/d.s a^lmi.ni^^ torcd Lo 70 soldic 
(■ r..u n*.''.' J wh(? h.ul coinr>l o tuvl advaJ^ced *:ralnLng in thf^ subiect lu/M. tcr. 
:^r<jrK-^t\ mac it ^ i^y t. iioso saiTK/: t,rainocr; ori a karui>j-on pcM forinancv? t^-^st. 
whi-(di had ,simi. lar i tcin.s re a 1 ^;o^ ol- tn'i i n^.^l . The ha^uis-oii host: wa:-'. 
admiri i..s t'r^rtsi r<v,it:inoly L">y the Army to all train<."'ef^ at tdu--' end of the 
advatu:rd i^raininq. A parallel y^ai.ieT^-and-pencil test wn; administered 
it) -i d d i. •■!■"- t fa ! Ill s , rmd iiari-]:-j-on '(^tps wet.T,' a.l :>o o])tained foi: 

th'W • t r \ ^ -s . 



1- ' [ ma. ; : 

'Viy r-n;ults rav'..red tdie f '.pis i hi ] .i. ty of tolovision to^>tinq. The 
t-.- ■ waf; :";rodiUCcd an-:i adn i. a i te r^xi w.ithoiit difficulty, aa>'l the examine^: 
hiaci a Vi^TV f a vc.-irahrl e atti.turle*. The Ln^anirv- had no trcubie^ under- 

tand, i.n.ry and resi'^oivl i to r:ho items. The exaiidnee.s iudqed the tost 
a.s "f.M.7.-" (impartial) i.n terms of t^;e;t: ime theni on important tasks tiiey 
sspald iiave ma stinted. 



'I^h.'' Vv\].idity of tiie losuilt^' v;as inconclusive. T!ic criterion scores 
for th" ha!Mis-r)n t(n^t wt.^re unsatisfactory in chat most examinees Tn:-'^,c 
a on-rf'^ct. score. Th-) correlation bet:v;oen the television and hands-an 
tests was lo" --pos i ti ve Ind: nonsiqni f leant . Comparison between the 
to b'":^v is i o!:! arid rjaraliel paper-and-pencil tests also showed no overalJ.- 
.1 i : fe r'-eict.? , al^diosrii tlu^re W'-'r-' sianif.icant di.fferonces between mai^y 
it.- SIS . 



EKLC 



Ml. U ] 1 M 1 



> f I ' i 1 n 1 i 1 1 ' I 



' ' ifiiT. t»-;t:; with t ( : i . ' v I , : n m i iitfMit-, n^plac^j !Mn.{s-(.n 

I ^"^ ' ■ ■ . 1. ■ ( ( M mi ii< ^ inoro fu^^'isfMy wlu'th^r t i ion 

' i 1 on t-st an. a nuirc t Iku-mu' fl i . ' xani i t i i ( i dI' {ho.^i' task;; 
t . ' 'I Mi; ;. > > ■< -riipi >Mi M )T ; ; t ha t , a{ jp* >a i [iio'it a;ii.'!iaM.' { c ^ t <M ^^V i Lon l^'atiiKi 



ERIC 



' ' ', J 1 t . i \ 



J M ' • 1 ■. ' W * n 1 1 : : I in! ■ ! t I ' . ; 1 . J 1 ' ' ^ 
)• . ; I • ■ < I ^ •■ ■ ^ 



1 ■ ■ I ■ •• I 1 ! ' .'■ . . . 

■■ i; ■ r - iM 1-- i^- ' i ! '1'^ ■ :t 
Ii 1- .. i: ^ M I ! ' I I ■ M i: L' ■• ' 

iV.\ I []' •■■ 

T < . ■< ' 1 ! M I 



M : i ; ' H 1 t ',/ f ' !. 1 ' ; 1 1 1 ' ; ' i\ ' I > ^ v i : ; i > ) n in ' r< 1 1 < i . 

> . t 1 : ■ v i i . ^ii 

>ni;MT") ^ > t' ^IN ■ I w i ; ; i oi i , . 1 v ipo t' -an<"l - Pf > m ( - i 1 r 'i'l' 
flantlr.-' in I s 



: 7 



TKLDVISION TESTS 



:>:AMrLi:s PAPr:R-A[^in-T[-:NCiL. r" 



KVAL[;AT OK TUP Tf-:T}-:STRATOR, PJQIJ.: i 



.3 1 
33 
37 



ERLC 



ii 



MS'i' CM'' tAmm:;; 

I - • 1 • r i pt i ' Ml ttn> '17 ittMu:; (^u t.hi^ n-!rvM:;'MMi t*">t. . 11 

^»<«'li'^in 1 L' <u]<i (1 i t r il)Ut irni < > Xiuni tuu v ; . |S 

^- MiMn i)t»rc;<.'nt. error ind-it* on ( Ik^ t ( ■ 1 ( ^ v i : ; i nn aihi })cip<M- 

.in(l-n« •nr* i I t ».Mt j / 

■I, Aii<ily:;i;, of vari.iiUM^ for l « M v i ; ; i c ^-n .ind ] >cii r -ai id - 

-r^ - i I { ( >.s t ; ; j ■ 

'» • Ml -ail Ml t .M^ror niaii'* by tln^ t ^ I - • v i : ; i i >im,] ^ . 

.ind-i w.' ! ), ' i. 1 cjrfuips on ilu' [ laiu 1: ; i t'-a , ! ' 

! "i ^int M r' i atjii in.d.v^M-n 1 ( d w M.^ j c ,)ri and { af ^ r- aiid - ja ' lu m 1 
it''m.'-> I "area ait » a rcu" arrarui^'d hy r( - v^n st - i\'pi'a, . . , 

rorr.dat mna l)td;w'.H^h tdio ta»Iovision and [ - j i t a;-arid - 
atau.-il s^'tart.^ and tdio 1 1 :)nc'ir)' ~c ui t f onTian< X' f-;c:ofaaa . . 

Id'-futa- l; L'onc/r^] t: Lon of '^tirnuliis fidality and f ra>; i hi J i.t y 

t: radc'Q f f y 

?. . 5-t.iaTiulus and ro.sponsc f idol i ^ ■ dii^iensions 5 

^, !vos[Xjnst:? com|:<ononts for taak : Lo^: Jiny round into 

na i a qun on an MGOAl tank i\ 



EKLC 



12 



TELEVISION AS STIMULUS INPUT IK SYNTHETIC PFRFORI-IANCE TESTING 



INTRODUCTION 

Over tiio past 10 years or so, tL Army lias tried to convert more of 
its tGstiiK} to the "hands-on" performance mode, especially at training 
center?"> and at the beginning skill levels. Even more emphasis has been 
placed on performance testing in the last 2 or 3 years with the begin-.. 
ning of the Skill Qualification Testing (SQT) program. Performance 
testing is highly desirable because of itr high face validity and high 
u ■ 'r acceptability; however, this type of testing is very costly, hard 
to standardize, and often not feasible. 

The alternative to hands-on performance testing has generally been 
the standard, group-administered , knowledge -type , paper-and-pencil 
test. Although relatively easy to produce and administer, this type 
of test is generally considered to have low validity and low user 
-^'cL^^ cep t ab i 1 i t y . 

Osborn (1970) has Suggested that a compromise validity-feasibility 
tradeoff point might be reached by using synthetic perfomance testr 
According to Osborn, the term "synthetic performance test" rL>fers to 
any performance test that is less than a full hands-on test, but more 
than the group-- administered, knowledge-type, paper-and-pencil test. 
Synthetic performance tests include all- tests that use any type of 
simulated inputs or responses. Part-task tests, in which only one or 
a few response components of a task are measured, are also included 
lander synthetic performance tests. The s-mthetic performance test 
is conceived as less costly than a hands-on test, but as a test that 
still ha-- reasonable validity and user acceptability. 

To support the Army*s adoption of performance testing, the U.S. 
Army Research Institute has iriitiated a broad-based researcn program 
to investigate the possibilities of syTithetic performance testing as a 
cost-effective alternative to the usual hands-on procedures- The goal 
of this research is to develop a psychometric b for botl hands-on 
and synthetic methods. 

The research focus har-: been on the use c audiovisilal media to pro-- 
vide the simulated stimulus input. The reasoning behind this focus is 
that audiovisual media stand midway in the' stimulus fidelity range, and 
at the same time, are at the medium to high end of the feasibility scale. 
Thus, audiovisual media may represent a good fidelity-feasibility tradeof 



"^Stimulus fidelity as used here refers to how closely the test stimulus 
resembles the real world*, and stimulus feasibility refers to how much 
it cos' ^ to present the test stimulus in a testing situation (high 
feasib ty equals low cost) . 
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Figure 1. Conception of stimulus fidelity and feasibility tradeoff. 
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point insofar a.'S stimulus input is concerned, Ficure 1 shows a concep- 
tion of this fidelity-feasibility tradeoff. 

The overall research^program has tlv' following objectives; 

1. To ox};lore t!he paratrio ters of the various audiovisual media v.o 
detex-mine the media's applicability to synthetic performance testing. 

2. To explore various responding modes and respor;se devices that 
can be used with audiovisual stimulus inputs. 

3* To determine whether those response comp( ':^,onts of a cask that 
can be measured using audiovisual media are sufficient to yield an 
acceptable measure of the entire task. 

4. To develop a task classification system that will enable a 
synthetic performance test developer to determine by analyzing the task 
(a) when audiovisual media should be used as the stimulus input, (b) 
which medium is advisable, and (c) v;hich response .components should 
be measured. 

Several experiments in this research program are now in process 
using a number- of different audiovisual media. This paper, which is 
concerned with television as the stimulus input, presents the results oi 
the first of these experiments. 

This first experiment was limited in nacure and focused on the 
feasibility of -using television as the stimulus input- As such it was 
concerned mostly with the first research ob jective--applicabxlity of 
media to tes t ing--with some exploration into the second and third 
objectives, responses to stimuli and test-task comparisons. 



Background and Ration ale 

The impetus for this researcli stem: from the Army's decision to. 
substitute the Skill Qualification Testing (SQT) progr^^m for the current 
MOS testing program as a means of assessing the job skills of enlisted 
personnel. The SQT program is intended to be based on job-sample tests 
wherever practical^ as contrasted- to the current MOS paper-and -pencil 
knowledge test. 

This change was brought about partially as a result of the research 
of a number of investiga s (Engel, July 1970; Engel, October 1970; 
•-:ngel fi^Rehder, 1970 ; . , j^^Key , 1965; Urry , Shirkey, St Nicewander, 1965) 
whc questioned the validity c~ tlie MOS test for job skill assessment. 
In 1966 the Army convened a special bo'ard of inquiry' (Erown Board) to 
survey the entire question of written MOS tests for assessing job skills 
and iob knowledge. This board recommended that performance tests be 
substituted for written tests wherever practical (U.S. Army, 1966) . 
Following the publication of the fxndings of the Brown Board, the Army 
has made substu.ntial progress in implementing the recommendation (e.g., 
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the Tank C.rewn\an Advanced Individual Training perf ormj;irfcG"t.ests admin- 
istered in--the form of a "county fair," with examinees moving ^from 
^•tcst to test around the exananation area, during and at the end of 
earrh training cycle). Flowever, due to high costs and difficulty i 
maintaining standardization, the pv^rformance test obviously is limited 
in terms of making up a substantial part of each SQT test. This is 
particularly true at the higher skill 1g\:^13 and for many hard-to- 
measure tasks. Occhialini (1972), for exa.,:ple, presents evidence that 
performance tests are extr'-moly difficult to prepar*^ and administer, 
and arc of ques.tionable validity. Engel and Rehder (1970) review tiie 
arguments agains- che uso of performance tests for part or ail of the 
SQT battery. Their general conclusion is that the exclusive use of * 
performance tests in an SQT batter^ \s'ould be too costly and impractical. 

/ 

Reacting to the pros and cons of pape r-and-penci 1 vs. performance 
tests, several researchers have proppsed comipromises . Engel and Relidei* 
(1970) advocate a mixture-of-measurement technique in each SQT test, 
comloining work samples, simulated tests, peer ratings, and paper-and- 
pencil tests. They present evidence indicating that cognitive items 
can be measured adequately by paper-and-pencil tests; that motor- 
manipulative items require v;ork sample or simulated tests; and tbiat 
peer ratings can be used to judge social, leadership, and overall 
abil ity . 

Osborn 's (1970) approach is concerned with developing synthetic 
tvsts that it is hoped will eliminate some of the impracticali ty of 
administering performance tests, v/hile reducing the verbal component 
and improving the validity of paper-and-pencil tests' Osborn visual- 
izes a continuum bounded on one extreme by paper-and-pencil knowledge 
tests and on the otjier by job-sample skill tests* Within this 
continuum, -a number of synthetic tests more or less removed from each 
extreme can be constructed. . The continuum is conceived of as being 
scaled in psychological units and varies along the dimensions of 
stimulus fidelity and response fidelity (or a mixture of both) . 

In j^ny combat situ^.Lion, the stimulus dimension would be a large 
complex composed of visual, auditory, tactile k-inesthetic /'olfactory , 
pain, and stress inputs''^ The response dimension would be an equally 
large complex of cognitive, motor-manipulative, and perceptual outputs/ 
For the purposes of illustration, the stimulus and response fidelity 
dimensions for armor crewmen might be co.iceptual^zed as shown in Figure 
2. Osborn maintains, in an analysis similar to the one shov/n i'n Figure 
2, chat one must pull away from each excreme of the continuum to develop 
synthe?tic tests that are both feasible and more 'valid than paper-and- 
pencil tests, 

/ 

An important aspect of Osborn *s concepticn is his reasoning with 
regard to part-task testing (Osborn & Ford, 1976). In this conception, 
each ta.^k is composed of a number of response -components divided into ( 
cognitive, ]:)erceptual , and motor behaviors. Figure 3 shows a task broken'^ 



4 

16 



i 



STi^^JLUS umm mmnA 

J / ■ I ■ i 



tir^=r't./"rr '^-cr 

' ' Picture's tquip^ienr Opv:ation:il UaiUUuu 



/ ' (Television] 



R.SPiroi. i-wnwi m-'s'.^n- 



i'e^nn:v;es llc^jvnso 



-Starr ■:-iiOi) ilC'^poiiM'S- 



\ 

— > 



Figure 2. Stimulus and response fidelity dimensions. 
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Figure 3. Response components for task: Loading 
round into main gun on an M60A1 tank. 
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down into response componGnts. 
on an M60A1 tank.) 



(This task pGrforiried by tlie loader 



riio reasonit-g beliind part-task te.stinq is that: it may iiot bo riGccs- 
sarv to test evory response component in a particular task in order 
to doterraine how well the whole task can be performed. It may be 
possible to get a good indication of whole-task performance by measurincy 
only a few ' response ^components or perhaps measuring only one critical 
rc'r^ponse component. 

Part-task testina becomes cruci.^i when audiovisuaJ stimulus inputs 
are used because the nature of th.e medium precludes obtaining any 
measurements on most mo t-or- response com.ponents . . - In order to obtain 
measurements on motor-response components one needs to test on real 
equipment or a haiids-on simulator. Since the measurable response 
components in audiovisual simulation are limited to perceptual and 
cognitive ones, it follows that the usefulness of audiovisual stimulus 
inputs is dependent upon the validity of the part-task testing concept. . 
One objective of tho research program is to check tlie part-task testing 
concept by correla::ing scores made on part-tasks using audiovisual 
stimulus inputs with scores made on ' the corresponding whole task tested 
in the hands-on mode. 

U^j of Te levision in Testing . Television has been used in testing 
primarily as a recording mediuiri (Cockrell, 1974; Hays & Pulliam, 1974). 
A study by Shriver (Shriver, Hayes, & Kufhand, 1974) explored the 
possibilities of using television as the stimulus input in a perfor- 
mance, test. After developing the test, Shriver concluded that telev.-5 - 
sion did not offer mUch promise in temms of replacing hands-on testing. 
He listed eight disadvantages of the television mediLim and decided to 
abandon the method and not attempt a systematic comparison between the 
television test and hands-on performance tests. Some of- the disad- 
vantages mentioned follow: ■ . 

1. Television tests place the subject in a passive role, watching 
someone else perform and evaluating the correctness of the performance. 
There is no reason to believe that success in this evaluation role.will 
insure success in the active role of performing the task. 

2. Television violates a major ground rule .of criterion-referenced . 
testing in that it emphasizes process measurement rather than product 
m.easurement- ' ' ^ > 

3. Television costs are'^very high compared to those of slides or 
graphics because of the large amount of equipment needed a^nd the large 
personnel time'* requirements. 

Shriver* s criticisms are informative, but they do not necessarily 
- settle .the case. The nature of the medium does^ include some practical 
difficulties both in producing the stimulus tapes and in administering 
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till'- t-.-'.:t.;; . ;iowover, those cl i f f icu.l t ios arc rrdnor comparod to tlio complo 
ta, i: <jf admirii:;; tor i ng hands-on pcrformanco tosr.s. If to 1 u'vi s ' 'ii can she 
a a.-cfal '•orrolation v/.itli job-saTnjvlo tests an^i also sh.ov; advantarTrMs over 
wrif t'-;i and other a\id.Lovisual. tor^tr.;, it ly e'oU iv.. worU'i t!if^' extra cor.i: 



erian^ii/v ohiectives of tn^^ present; r.xr^'' r i;;K^nf: wer-^ to ap^'rais.- 
■•■'I'- t:i':» practical, difficulties in nsinq television as the sciinulus 
. . a [ > d t: o ni a I: c a r o ii q h c o m p a r i i: o n a mo n q t 1 ^ v i s i o n , p a ] c- r - a n c 1 i , ; i 1 , 
'.ea d M d a ~ o 1 1 p ^ r f <:> r" m a n cx* t o t : ■; . T } i u s e c n d a r y o b j c c t i v e v; a t c • c o i r i u ': ^ t; 
a ' • ■ • !•: c; a t: o f a r < s p o n rU n q 'i -a vice ( T e 1 e s t r a t o r ) d c ;" ' i q i ) ( j c ] t o j i:^ r u i i. : 
. to rt_^spond directly to irnacjes on a r-;crera) (sc.' Apc-adi-X (') . 

"a." ^a. ficLilly , t;u^ orriective^; v;r^re as *"o] lovs : 

I . Dc- 1'^ r m i n e t h e- f e a s i hi 1 i t:y o f s j nq t a 1 o v i a i n in " a ; i a . Ti : 
i r . • iTi; : u nd • J r coi^ s id e r a t i on he vc ■ v;e r o i ' 

o . Ta de rs t and abi I i t y of test i t ems 

'a. i:ase of respondinq 

c. 'rime allotment for respond iiiq 

d. I)i f f icul t ioo and costs involved with adiTdnist(^ring tele- 
vision tca:ts. 

determine the acceptability of television testing by examinees. 

^. Comp-are tI)o results made on the telc^vision ast wit! ^h-^ae mad(^ 
on th'.- paja^ r-and -pone il and hands-on performance t ;ts. 

^1 . Con due t a che ck on t of the Te 1 e s t r a to r re • s po n s e d^ ; v i ct • . 



METHOD 



The overall method consisted of (1) producing a television test for 
a sample of 'tasks from the job field" of tank crewman (IIK MOS) , (2) pro- 
ducing a parallel paper-and-penc i 1 test covering the same items, and (3) 
com])aring the results made ona these two tests with the results made on, 
an existing- hands-on performance test that covered many of the same items 

Tfie job field of tank crev-nman was selected because much prior 
r< 'Search had been done in this field. A complete ta^sk analysis was 
available, and a hands-on performance test has been lai use for the Tank 
Crev.man Advanced Individual Training course for 2 years. This existing 
hands-on performance test was felt to.be a good base against which to 
compare the television and papcr-and-pencil tests. 
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The first 5Jt.ep i.n producing the television tape was tp select the 
critical tasks in consultation with military experts. The selection 
criteria werf set by the military and included such considerations as 
importance to fulfillinq 'r.iic mission; safety to the cre^nan, and safety 
to the ef]uipmont, Tiie critical tasks selected were quite similar to 
t:v^ t:asks> co-'erc-d, i.n th/,- Tank Crev*Tnan Advanced Ind.ividual Trainin'r 
courst.'. After rho critical tasks were solf:^cted/ they wer^.^ ordf-'red 
ac\.:ordLP/j to skid.l 1 vd. . 

Pr>r tiuj firij]. '.tjst^ ^-.asks wer-.: se ,i.^_■c^:ec from skill levels 1, 2, anc 
3.'" L'or thu \n\\:vi ■i./s of this cxf- -r ini-.-nt , thu'3 tasks can be consKjered 
to r"U"iqe fr')n: lairjy easy to ver^ . i i r cu 1 1 .. Tasks were also suiectL^d. 
sue!: that k\xcY\ ...f tiie four r^o' ..ions (driver^ load(ir, gu!iner, and tajik 
C'.:r;:nand';i: ) was cove]:L;d, a:: i few uasks [Ajrtamed to t\)0. crew at iarqc'. 

In )ri:uii ta t ion witii r-h^} military, eacii task \y\s broken dowr"'. into 
::o'.;:. i tiv*?, p^^rcep tii.. , and motor components; and each rv-sponse component 
W'V" •■xamineri for its .■'siticality r.o thf? task. Practical considf^ratioiv- 

.si; as ov^>rall tf:;:^>t riisninq time, time to televise each item, number of 
i'-.spoi'ise coniporusits needed to cover a particular task, and achi.evinq 
a balanced test (see Apr^endix A) eliminated many critical response 
rompon(.'n t:. . Por each of tiie remaining critical ";ponse comp;onents a 
television test item was conceived and a television shooting script was 
written. Each itemi was televised in a crude fashion v;ith a handheld 
'.'■imera and a p( !"table videotape recorder. 

The raw footage was edited roughly into a prototype television test 
by ttu: addition "of narration and titles,.. Tlie prototype tape H^as intended 
only as a model for a professional tape to 1" > produced later and as a 
vehicle to ch.'.'ck technical accuracy and television feasibility.^ 

Military experts checked -the prototype tape for technical accuracy 
arid uiiderstandabiliry . h revised ':elevis]on script incorporated 
sucjgestions; a final television tai)e was produced using professional 
television p'ersonne], t-am.eras, and editing facilities. The shooting ana 
editiufj of his final tape required approximately 30 calendar days (about 
15 actual working days) . 

T'he final tape consisted of ^1 rest items plus 4 p^^ar^ice items and 
iiad a running time of 5 3 minutes. Tlie items ranged in running time from 



^There are five skill levels for each MOS ranging from skill level 1 
■ (beginning) to skill level 5 (most advanced),- "^1^- 

. * ^Work on the preliminary television tape and the task selection required ^ ^ 
FR?r^^ produce it were done by Human Resources Research Organization under 
HsiMsa contract to the U.S* Army Research Institute. 
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Table 1 provides a description of the final telGvision tape. The 
riatcgorizin:: of response components into perceptual, cognitive, or 
• motor, types v/as somewhat intuitive. The intent was to show the pre- 
dominant element of each respotlse component dnd not to imply that 
other elements v;erc not present. . ^ 

Ot the 47 items shown ii": Table 1, on?y 37 were administered 
to the (examinees in the experiment and only 30 were scored. Most skill, 
level 3 items were eliminated before the ^.^xperiment upon the rer .mmen- 
dation of the i^iLlitary staf" at the Armor Center. These items were 
considered too a^.;.ancod for the examinees^ After the start of the 
experir it, several p.::. r:ary advisers recommended the elimination of 
six mor-- items, and one item was eliminated d- ■ to a peer ' teleirision 
'")icture. These seV' !; items vjcre administered but not scored. The 

/Otnotes i-n Table 1 qiwi the reason for the elimination of any item 
and also explain wK^/ certain it- m:-) were not included on the hands-on 
test. /" 

A more s. 'jcific (description of each respOiioC type shown in Table 1 
follov;s'. 

{!)• Multiple clioicc. ' The examiiiee was required to sel'^jt one 
answer from a list of three, four, or five alternatives. These al- 
ternatives were sometimes tlie Sc e as those in the usual paper^and^ 
pencil test — namely, ^ words on the screen~"and sometimes cohsisted ,of 
in. jes on the screen. 

(2) Error detection. . The examinee was required to watch a procedure 
being performed on the screen and to indicate the time anc? location of 

an error, if one occurred, at the time it occurred. The examinee was 
shown the procedure twice and responded on the second showing* 

(3) Motor manipulation. The examinee placed a plastic gun retitle 
(those fceti.cles used with the main gun in the ',M60A1 t^ank) on various' 
stationary and moving targ^^ts as' if preparinqyto fire the main gun. The" 
reticles were also used to simulate the ad^};«fstmGnt of fire that would 

be made if the first round missed the target. The motor^manipulation'*^ 
. \ response was supposed tv» be a'Ncrude simulation of the actual response 
in aimincT the main gun. However, the movements required were so far 
down on the scale of response fidelity that the motor component appeared 
not to be measured at all* Perhaps the reticle response was primarily 
* perceptual and cognitive. ^ 
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i. t'.'in-L)'/- i t om l)asis. The stii.mulus inpu^ nn this test v;as prinarily 
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ita-nva Wi.'ia- adii: i.n i .-; tor- aiu'l r-nly 30 wora scorod., Tlit..: i t.r-inr. :-:("'or<^-a 
'.v-^r-/ tiio ^-.a: as Llio:- . sccjr • for ' * t':^lovision t'ast . 

T: I'- I >a[ 'C r~ ai i-i - pua c .1 1 i tcrriG a I'.d th o to ] a vis ion i tems d i f fa rod ' ; r i * ■ 
i : I t he: a mr, u n t: o f t i m j a 1 1 o i: t o d to r ^: pond t O a a c 1 1 i t o r n . Tl t c j a 1 1: i r •. 
1 irnit: war,; t\\>.- .saiao f(jr l.Ajtri tests; l^owovor, examinoos coul.I a 1 1 ocm t*.- 
thv' ri^spr.nso timo any way tJioy choso an the papio r-and-porn; L J rnro lar 
•.vo r- • r'.">s t r i.c t^ 'd, to 10 s;ocond.^- r Item oi"i tlic t o 1(? vi . . : ori test. 

■ ni t:u,r papor-cind-i-'oncTi 1. tost, c^xamrnoos could c' riqo tho i r aan s'a^'- rr-- , 
, ^ >: L p i t o m.s a r ^ d an s w » ^ r 1. n t o r , n d r o v i o w t h o i r a n a wt r s ; o i: i ) o t v': 1 » : : - 
V i a i. c)i ^ t ' aU". , 1 1 <a n o o f t i i i s' f 1 c x lid 1 i. t y w as p rm i *; t i : ,. 

'1 1 1 -s- s r.^ d i I I • r o i i c o s h<j t v.'o o n t : i o t wr- t o s t s wo r o r ^^ t a i n o d be c a u s ^ ^ c a c 1 1 
inodiuin londs itsolf most roadilv^ to the typo of procedure used. Any 

■ a^jr procoduros or a common procedure t or both toasts would ha^o roquirr 
muc:i mc^r^i control and tii» rehy r^^duced administration feasibility. 

Ilands-On Performance Test 

TliG hands-on test was cno^ routine Ly administered to tank crewmen 
trainees as a final examination for the Advanced Individual Trainin-- 
arr^or course- This test was given in the form of a county fair witn 

ations and 30 performance measures. Examinees were graded on a 
"go/no-qo" l:)asis for each performance, inoasurs'. r'or each no-go, exam:'n^:a^ 
were rt^guircd to seek out remedial training a: ; report back later for a 
rotost. If tlie "re test was a no-go the exami; had to report back the 
next day, after further remedial training, for .second and final test.^ 
For the purposes of the pro sent (.'^xporimen t , the score recorded for ^ach 
exam.inee was tlie number of first-round no-go's. This was not a partic-- 
ul^^rly good criterion because the number of no-go *s was very small*. 

He spon s e Egu ipmcn t 

, A secondary ob j ecti ve of the study was to check out the te levi sion 
r(.^sponse equipment (Teles trator) * This equipment consists of a cl<?ar 
plastic electronic tablet and associated recording and programing 
components". Tlie electronic tablet covers the television screen ,(the 
t^ablet is approximately S inch away from the screen at the center of 
t?ie screen and approximately,! inch away at the edges of the screen). 
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The examinee looks through thu ; ablet to view the tc items. Kosponsos 
are made by touching the lac^? of the t^iblot witli an Glcctronic stylus or 
an electronic gun loticle cit a particular time and location. Before the 
te:-^t, the correct answers , L-ime and location) are programed on the tele- 
vision tape. During the test, examinees are credited with a correct an- 
swer if they touch the screen at the correct preprogramed time and loca- 
tion. Any other response Ijy an examinee is counted as incorrect. Only 
one answex* is' permitted for each time, and the first ansv/i.; r--corrof t 
i.ncorrect — made during the 10-second response period is counced. 

The response equipment was in prototype foiip and bee ise of opera- 
tional difficulties could not be used for the f >:periment. However, i-f 
proved possible to test the operating concept of the equipment by placing 
a human grader behind each examinee and having this observer record on a 
sheet of paper whether the examinee touched the correct location at tl-' 
correct time. This grc. i.ng task was quite simple, and during a pilot run 
with eight examinees there were no difficulties in grading. 

The television monitors were black and v/hite and measured 1 5 inches 
diagonally. The examinees sat approximately 2 feet from^the sets at 
self-regulated distances so that they could manipulate the resp(5n;^e im- 
plements comfortably. Prior to the start of the experiment, it waV^^de- 
cided to remove the electronic tablets from in f -r-int of the screens be 
cause of parallax problem^' . After the tablets were removed, the accuracy 
of the responding and scoring improved to a very precise level. 

The response implements consisted of a stylus used for all multiple- 
choice ^and error~de tection items, and two plastic gun reticles used for 
Totor-manipulat ion items. The stylus was simulated by using the eraser 
1 i an ordinary lead pencil. The two plastic gun reticles, the same 
desi.j.. the M32 and M105D main gun reticles in the M60A1 tank, were 

manipulated by small wooden knobs glued to the plast.LC reticles. 

Examinees 

The examinees were tank crewmen who had just completed the Advanced 
Individual Training/Armor course. Altogether, 134 examinees assigned 
from three different companies were tested. Examinees were drawn from 
the companies by a selection process best described as haphazard rather 
than random; hov/ever, there is no reason to believe that selective bias 
was present- As each group of examinees arrived for the experiment for 
each session, the group was randomly assigned to the television or paper- 
and-pencil test. Originally, 144 examinees were scheduled for the ex- 
periment, but 2 were lost due to scheduling problems and 8 were lost due 
to scoring problems. 

Procedure 

Testing was conducted over a-^ 5-day period in three morning and five- 
afternoon sessions* The actual schedule and distribution of examinees 
are given in Table 2. 
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Tablc.^ 



Schoflule and Distribu! i.ou of Kxamiiioes 



•iV^st and gays [ 

timo ot: day 1 2 3 ^ 1::^".^ 



To Icvxsion 
Morn ill CI 



16 IG 16 il 11 70 

8 -8 B 24 

Aff-crnwon c< R B 11 11 46 

pencil IC- 11. IB 11 8 64 

Mornin B 4 lf"» — 22 

Aftori:->or^ 8 7 11 ^"^ 4 2 



r:ach group of subje cts reported at ^0800 oi 1300 and was given an 
oriontati i session explaininq the purpose of the experiment. All of 
tlie paper-arid-pencil group was administered the paper-and-penci 1 test 
right after an orieri .ation , The television test was administered to 
four examinees at a time; the rest the television group was assigned 

to a waiting room. Both the telGvii>Lon and the paper-and-penci 1 tests 
required approximately 1 hour to complete - 

AD^^roxim.i . 10 minutes of training were required to teach the 
o:.aininees tho m. thods for responding to the television items. Most of 
thiF training was concentrated on the use of the plastJ c reticle. The 
examinees were trained by having them respond to tliO four practice test 
items. rf any examinee had dif^^:ulty V7ith the reticles, such as choos- 
ing the incorrect reticle or holding reticles incorrectly, the tape w'-s 
stopped and the four practice items p'resented again. Tn no case was it 
necessary o present the practice items more than twice. 



RESULTS 

Fe a s i D 1 1 i t y o f Using Television in Testing 

Th c:..iminees did net appear to have any difficult':- in understanding 
the it- rns. All of the ^ ntent had been covered in the Advanced Individ- 
ual Training course r and the examinees had been tested on similar items 
several times. 7^11 of the items were also performance based and posed 
questions that ur normally in everyday operations. 
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Tlie rt!ispondirig proceoded smoothly for most items. T'no examine^^s 
rt.spondod very inickly on the oasy items vapproximately i-2 seconds 
with the stylus, 1-4 seconds with the reticles). On difficult iteinr;, 
the arxT-^urit of resi^oii: e time allotted (IQ seconds) still appeared aiui-'ie, 
althou^ii thert, u uially would be a lot of hesitating over the ansv/ers. 
On only a few i.tems the examinee.^ failed to rei;i|.>ond. When queried 
after the completion of the test about the amount of response 1 : , 
iTKDSt e>: 'wiinees indicated that for thie most part the response time was 
adequa A few examinees sai that more response time should have 

been lotted to some items. 

The administration of the television test was more time consuming 
than that of the paper-and-poncil test because of the need to provici- 
pr(-l iiriinary training i the correct waiy fro respond ar^d the limit ■ mV 
four examinees per session, Adminis trat-i on could be made more asibie 
by incrcAs r tiie iv inber of television rnonitors, but ^ "■^'""lUld still be 
advisai- lo have ono test administrator for Oc^ch four .^iriinees because 
of the examinees* unf amil^iarity vvir.h the response method- nared to 

, t\\o administration time for hands-on testing, however, tc s:ion testin^ 

i.; much less costly. 



Acceptance of T e .lev^i sio^ Testin g 

The reaction of the examinees to t';c television test appeared to be 
quxte favorable. Postexaniinat ion interviews indicated that most ex- 
aminees actually preferreci the television rest to the hands-on test 
and all examinees thought the television test was fair. Even when 
queried about the test's being used as a bas ^' ^ ^or promotion or extra 
pay, the examinees still thought it was fair -»me examinees pre- 

ferred rhe hands-on mode of testing, but no ^ ; ^referred the paper- 
and-pericil mod-:^ . 

Some reasons ment. -^ec^ for preferring the television mode follow: 

1. Scoring is fairer and not dependent upon the whims of the test 
administrator. 

2. Testing is faster and not so drawn ouu. 

3. In television testira no one is shouting at you and ordering 
you a^'ound - 

Some of the reasons for oreferring the ba.pds-on mode follow: 

1. There is more time to think "nd to respond. 

2. Testing is more sprec^d out and doesn ^ t come so fast. 
3 . Television nurts t^^e eyes . 
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4. 'riHTc i.s c\ ^ \A\\av. l-.o walk arounci between items. 

The .examinees ai^-Jo indicatod that- tGlovision testing Vv'o\ild bo 
be'ttor liian papor-and-ponci 1 to^cinq bec:auso the questions would bo 
more viii ler.standablt ^ ^ r.-:iquirG Tnucli loss reading.. 

Com].^ai; ison of To leviiiioji , PapGr-and-Peiici 1 , and Hand.s-On Tests 



The comparison between the mean percent error mado on the television 
tost and that made on the paper-and-pencil test is sliov^n in Table . 
The means for the television and paper-and-pencil tests do not diifer 
to any crreat degree, indicatina that, the difrficulty levels of the two 
tests cire fairly equal. 



Table 3 

Mean Percent I^rror Made on the Television and Pape r-and--Fe no i 1 Tests 



Tost and r.).ays 

tim- of day 1 2 3 4 5 Mean 



Television 

Morning 10. G 3 15.00 14.7 5 ' 16.46 

Afternoon .^V.hw 29. 3c^ 28. uO 26.0^^ 27.09 27.34 



Unweighted mean 22.00 



!^apt^r-and-i.)enc l 1 

Morning 28.38 20.75 22.3 

'Afternoon 27.-13 24.71 . 5 29.90 27.38 



Un we i gh ted mie eun 



One interesting facet of the data is that afternoon television 
exayp.inees made m.anv more errors than th*3 mor'ning groups. These results 
are e .nvincing because tiiey are consistent across the first 3 days of 
the. experiment and because the afternoon means for Days 4 and 5 are 
consistent with the other afternoon means. There c es not appear to be 
any morning-afternoon effect " '^^r the paper-and-pencil test. 

Th.- lalysis of variaiice using the unweighted means ana.iysis for 
unequa_, cell frequencies (Winer, 1962) "is shown in Table 4. This anal- 
ysis sr.ov;s no difference betv/een the television test a^xd the paper-and- 



24. 23 
26.05 



25. 14 
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pencil Uv;t i r» tt-T-ni:; '^T i ■ ' Mn c ! i T j fli ). t; y . Tli-^rt' w.i:. .' ; i cjri i. f i < \ in t 
morn i t ' ■ T'Hi )v n\ *>\'i'-ri'\ , \)\\\ t nuwt^ nio<ui i iv|f ul ri^'Uilt \ r. I. } u - 



i t ni i f i ^ \ it M iiH '<"Hi >;< [iM ' ( M^i ) i u I t ' r<\c lion, Ai m I 
iic'ti(wi rcvt^cil:; thai tfn^ riK^rn i luj-a f ^^'frux^i t'ff^'ct 
tin- n ' b i "s 1 f 'M t (.•;:{ anci not on i lu- pap<^>--- -aul-pt^rv;: 



! ; r ) i" this -^l i. n t i a- 
: * c :( '>n a 1 1" iMt » 1 c ai 
1 t taat: . 



in ordiw to (^a(.a:k on whrth'.a/ t ht.^ a f t a^ rau")r3a xamiru.a^s iliay li.ivr' b<}(aj 
ba-:;; qn i 1 i t' i (a.l tlian thv n^Mai i ru] oxaininopa, th*' firaa: rDuncl no-~qr;*s 
l^foin (•:■, hand.'-; -on tri'^t vv^ a't ■ analy:.anl. 'j^ioac^ roaulta^ arr^ shown in Tab! 

. T nj-:p» a.; t. i 0[ I of t:lu^ mcari'; iridia;ata^' 1 i t" t. 1 a cl i f f o r cai f ^ b(Ma,v;^,?on t ho 
r a 1 f i : ■. i on tUi' l [ai f )(^ r- and-f Hai c i ,1 < froupii, o)" liotwcaai tdio morn inq and 
«.i ta"^ rnoor'j ciroupa. If anVLiiinc;^ tdio a fta.^ rnra.)n «.^roap porformaai sli<Tht ly 
b« a t o r than t:h»- nv)rninq qrcniiJ. analysis of variance of td^oso 

r«^stilts sh.cv.vod s i »-?n i. f i c awj t d i f f e raaico for any of thu varia^;lo^:. 



■rablo 4 

.'\ n a 1 */ : d s c d" \ ' . j r i .^'i n c a f o y- • 1 v i s L o n a i ^ d P a \ ^ o r - a n d - FV"' r i c.d 1 ^ t 



.;>au.ao ' df MS l*' p 



TV vs. : . I (mr.thod) 1 0,S0 1.04 n5^ 

'orr' nq vs. aftornooTi (staasiaaO I i:?4.57 13.^,0 ■ .('i 

I'-^aii'^Li X ■ ss i .1 . bb o.4a . s'.. 

VJi t!d n. col] 1 30 9 . 16 



Althioucjh overall scores on tlie titjlcvision and pape r-and-penc i 1 
tests did not differ, there miqht be differences amonq the various 
items. Accordir.gly , the items were grouped by response type (multiple 
choice, error detection, and motor manipulation) and log linear 
Chi -square tests {Shaffer, 1973) wore comnnted for eac'n item- Table 6 
shc.-.-,s that there was a wide variation of dd.fficulty amoiig the items 
ranoinq from 1(~) to Si"^- eri'or. ^"or t]"ie m.ultiple cho ice — i terns , there 
v;as I I I 'de difference bet:ween the television and pape r-and-penci 1 
versions. '"-^ily 1 of 13 J. terns showed a significant difference. For 
the errors-detection items there was a sul>'- 1 an t ia 1 difference, with 
s.LX out of nine items showino a significant difference. 
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M(Uin [\.^rcoiit l':rrc:kr Made by rht^ 'IN.vl c^v i : ; i (~)ri and 
Pcii^'M ,ip.:i-P(Mun 1 Or* s on thi:' nands-<'"'5n Tost. 



Tt^st and 
tinK^ of day 



Days 



Mean 



To lov i 5:; ion 
Morn in q 
A ff tor noon 



. 00 



(> . on 

O . 00 



9.5 2 

(\ . 5 2 



0 . 56 



9 . or> 



8.16 
6 . 5 2 



Papar~and~penci 1 
Morning 
Afternoon 



Unwe Lcfhto*.! moan 



i2.00 0.00 8.00 

7.52 9.72 5.52 12.00 



6.5 2 



7 , 34 

9.08 

8 . 48 



linvici qh tc^d moan 



8 . 78 



It is intftres'tinq to note that errors of commission are raore diffi- 
cult to detect on television; whereas errors of omission and no-error 
Items are more difficult to detect on paper-and-pencil . Three of the 
eight motor-manipulatio; ; items show some significant difference, and 
all three of these item.s show more difficulty for the television test. 
The net result of this item difficulty analysis shows five items more 
difficult on television tests and five items more difficult on paper- 
arid-pencil tost~s. Thi.'=; canceling effect is reflected in the overall 
nonsignificant difference between the television test and the paper- 
and- pencil tes t - 

The last analysis, in Table 7, shows the correlations rf the hands- 
ow est with paper-and-pcncil test and the television test- Those 

correlations are also broken down for the morning and afternoon groups. 
There is a lo^^ positive correlation ipotween the television and hands-on 
tests and also between the paper-and-penc il and hands-^-n tests. The 
paper-and-penc\l correlation is significantly differe , from r.erO; 
however, there is no significant difference between the television 
versus the h£md?-on and the paper-and-penci 1 versus hands-on 
correlations. The breakdown for morning and afternoon groups shows a 
somewhat higher positive correlation for the afternoon group and very- 
little correlation for the morning group . Once again , there is no 
significant difference between the television and paper-and-pencil 
correlations with the hands-on test- 
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(/oinf>ar i iicjn n(^tw»;v.Mi lov L.^;j on arid Paper -am l-F^(.^ru.:i i 
1 1 (^mi5 Pt^rcorU: T^r t'c^' AriMi^H'* t b'/ l-^'^sfion:'!? Types 



item numbr^r 



TV 



r rot: 
K I> 



Mu 1. 1 iple o>\\c) i •:. . ^ 

.1 

"> 

A 

7 
R 
^) 

1.1 

1 2 
17 

3:: 

error d(,^ t-^ct ion 
(commission ) 
3 

13 

14 

37. 
Er >ror 
(omif. 

IG 

2 3 

Error detection 
(no error) 

15 

20 

Mo tor man i pu 1 a t io n 
(reticles ) 

19 

21 

24 
2 5 
26 
27 
28 
30 



lOtect ion 
-ion) 



i 

4 3 
2 1 
3H 
47 
2^. 

ir, 

17 
3 
0 
1 

81 



51 
1,1 
40 
4G 



19 

4 
13 



16 
1 7 



46 
51 
4 
27 
11 
14 
23 
2 3 



0 

1 4 
7 2 
5.! 

?Ci 

1 6 
[b 
D 
3 

7 7 



28 
11 

12 

56 



47 
45 



19 
66 



16 
17 

5 
11 

2 
11 
34 
23 



IKS 

L . 4 3 
. 82 
1 3 - 83* 
ns 

n s 
n s 
ns 
as 
n ^; 
ns 



7 . 47* 

ns 
11 , 09^ 
1 . 42 



14 . 88* 
18. 90* 
11- 26* 



ns 
30. 36- 



12 . 24* 
16-17* 

ns 

- , 75^ ■ 
ns 
ns 
I . -4 5 
ns 



01 
10 



20 
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r.' • 1 .iL. 1 }^(^l.woi.M; t:h(^ 'IV:^ U.^v i r, ion <\\\d Pa | u' r-at k 1 - 1 "n c i ] 



Over a 1 I 



'li ' V i i-on V:'.. iutiids-on .^'-l as 



. ii ff (Ma^nca^ i^; tuM"iSLqrii r.ir:antr. 



Mo ri wi aq 



TolevisLon vs , lian'lj.>-oa -.0'^ n:; 

Pav>er- anei-t H Mici 1 v.*]. lianda-on .1*^' ns 



fferenctj aons i qni cant 



Af Lcrnoon 

Tolevi.:-;ior^, , liands-on 
Papor-anci-[>:^iic i i vs. riancls-on 



4 7 . .'01 
40 . .01 



d i f f o i; jinco i aon s iajn i f i c an t 



CONC^I AJS ir^NS AND DISCUSSION 

!:ie results froni^this rosearch indicate that it is ^-ossible to produce 
a. s'^'nth':^ t- i c r^r.t nsin^ telr>vi5=;ion as the st.imulus input. The examinees 
can unders ,iid the problems, make proper responses, and ac(.;ept the test 
as *'fair" for career evaluation. 

The experience gleaned from the production an^i administration of this 
protot:.-.>e test indicates that television testing is more costly than 
paper-and-pencil testing but far less costly than hands-on testing. The 
production of che tape, from conception to final editing , required several 
months and u^73d the services of a sobstantial number of professional 
people. Television tests are also somewhat inflexible, not only in the^ 
difficulty in oifectiag changes in the test, but also in the timing 
decisions-— tlie amounts of time to allot for posin each question and for 
each re s locr. se — th at h a vc t: o be made be fore t he p r odii ^ i on of the test. 

Television testing will a.ave a much mor'e promj-sing future if a 
presentation and response device can be designed which will perTnit the 
examinee to advance to tiie next item, as sooq as the present one is 
answered, to see tlie same item twice, to chaYige answers to an item, and 
to review the entire test. Such a capability \^ould permit the flexibility 
of presenting multipart items, such as in troubleshooting and would per- 
mit the presentation of multimedia items, such as using both television 
and technical manuals in the same item. 
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T))o 'Son t. <.^xj n ■ r.i iwi) I". pix>v i ch o v i dt mil"',^ t h< 1 1: f * ^ 1 * » v 2 . , i ( »n l.'^'.; r .) tuj 

IS In.qlily a(:r;eptc-iiUo to Lht.^ c^xami iv s.; . TIkm r t ' r-( '« ^mi r lan t at 1 :i tudi^ was, 

I. hat: Lll^' tost v;as littl'* clirrt^HMit frc^nn tin.' haiuls-oTi to::!.:^ in t lu^ Ad 

V'm<..:sd Ind.iv1du.1L Tra iiiinrj c.\)urso , csxcn^pt. 1 h.it. t avI < w i : ; i < was (lui t'k(M. 
an^.i l(;:;s, .s'ub]Cct. to scoi ing error. All c.:d' tlu' ^;spnt^s wimt: 411 it(^ famil- 
iar ta.) tlu- cxamitUMs;, and t:lu} j r.PiTV.; were onev- t.hcd tlu-y liad stihiv^- 
iru^ fcu" JJ- 1 3 W(u\ks, . 

TV' i f^v.i.:; i on u.S'd in tluj imi 1 i ij) lo- d 1 < ^ lorinat apf^oar- la. o f ■ i a., 
advant-.acn" over s.licii^ nr iu\\nn~-c\nc\-'pcM\L: i I formats.. }icfe->rt? fh" i-xy)cr'^- 
iTH'nt, it: was f o 1 t that: t.rsl ovis i on would c:.ff(M:- an advant:,iq(- f\ -r t:lio5^o 
if<.!!ns in v;h.i cli mo1;ior^ wa:^ an .int.oc/ral part, t.^f t\\r i: Imu I us . Vor «':-:am- 
[d«', S}>ancT^.uiburq (l'.^7^) has sliov/n th<it wat-^.'hiM'j a tasJ <>v.l s i on display r-.v 
a i.>roc'jdurt.' involvin<j mof ion leads to uuiro loarning fhan wat:shint:j a 
scquvru.'o <:)f still shc:)Ps. \U?Wi:\/"r, this advanf..Uj(] of mot:ioTi proved t:o 
ho true fc^.r on(j mot:ior! item in the r^resont r^ssoarch (item G, Table b) 
but not ^-as.ie for two items (5 and 7) . Perhaps if more motion-tyjx^ 
vtem.; tiad botsi included m the mu.l t i.j do -cl io i ce eatCKjory an advai)t:acjr^ 
micjlir Juavo bc^eti r,]'K>wn, 

Tn tlie error-de teotron cateqory there did api)ear to be a c^bsar-eut 
difference bei:weon telovisron and paper -and ■-[■)Gnci I item:;. Hert.^ the 
fidelity the stimulus did seem to play a role, and the enriched 
stimulus of th-e television pict:ure may have presented cues> to the ex- 
aminees, ThiO two V- J. ror-de toot ion i; -''Ps tlieit proved to be^ more difti- 
salt for 'elevisi.on exarri.inees (iter = ■> and 14, Tallies 1 and 6) v;ere 
two of the first orror-detect ion it-sr. . to be presented. Since error 
detection was an unfamiliar response for the examinees, this unfamil- 
iarity may have caused some difficulty. This same p^ienomenon can bo 
seen in tlie motor-manip.ul at ion items which involve an even more unfa- 
miliar resf^onsc- Here tiis television examiiiees had more d i f f i t\^ 
with the first few items tj^an did the paper-and-penci 1 exam.mees, 

Th.e correlations betv;eer. the synthetic and tlie hands-on tests are 
too low to warrant recc^mmending the sunstitutior of synthetic for hands- 
on tests. However, the correlations for the afterr.oon groups are high 
enough to encourage further research. The hands-on criterion test used 
in the present experiment was somev/hat unsatisfactory because of the 
large number of perfect scores. 

The drop in the scores on th.e television test for the afterI!^-on 
group as cnmr »r- I Lo the morning group was interesting but unexpecb d. 
One possible explanation for the drop may bo that the exair^inees wer 
required to stare continuously at a fairly large television screen : rom 
a very cluse disrance for approximately 1 hour. A human being may ..e 
able to tolerate this strain in. the morning but bv the afternoon accumu- 
lated fatigue plus a heavy Army lunch may have corribined with the stra::^. 
to produce a letdov/n. 
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'I'll' > M t this study rav( > -t I 1 hi ^ i'f m v. i 1 i t y o t i • • 1 < • v i ' ; i < 

t I ; t i n<f . Tl b ' ! t v- t i; t \ ■ n v ] > ro( t lu *< v i at «i r t si: ;f )r nt » I * - ( : t , i ' 

.u im i I] i : ; t t • • J i ii rr.i :u-.m 1 maiiiKM*, a?*' uiui< ■ r;-.t aixiah S • 1 1\' tin' *':<<iMMfi«' 

ami }mV( ^ 1 1 i ' ih a. ■< '| > t , i 1 i t y i t h t ht ^ t.^ x/i mi s i . 

validity d" rl:*'' Tt'sult.^; was i lu -c ^n. * 1 u s i v( ^ . 'V\\' ^ ^aitf^racui sr-ru'"- 
f<,r tlh' h<uuls--(>ii f'.t W'Ti^ Mtis>ci t T s far: t t a y in tli.it i-- ' haruis-fwi imiiK 
ma(l<' t'-.t-f(M'r S' a^'. 'i'li-^ C(;>Ta-(.Mal ion ludvy. tai tiu^ -^vi.sion ami hand^ 

l.sMs was 1 Mv; p(Ksiti^a- l)ut luMiS i i f i t 'ati t . f V ni'ij m > i ■ a i l>«-;v;tMM! j^aradl.' 
t I • 1 , ' \' i : ; 1 <.-ni aiwi papoi a • id~ [ )f a m. * i 1 t-..a s t: ■il'U) sli<>W('d nci d i f t ra ^ aiav ^ or. .tn 
ovt ' r . ! ] 1 ha.s is, a M a^ )U' M i t 1 la I't ' v;i ■ r"* ■ s i qr^ ' i i. c v i a t d i t ! ( a"' ai f • ■ ■ t io t va '^ m i 
rnai i \^ i i • - inr; . 



r:vidaai',. td f >m tins .^■tady is i n s. a t" r j f a » ■] d. > foiKd^d. 
[ M a d"i": r"ia, i [ n 'a r i - s } : , with ' > > 1 r ^ ^a s i ( 'S i nps t ' ; a,\M I >^ ^ a s^ 'S t < > 
r oT ai : -a t < - s( s . 
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ln'\- . ; 1, , and and Mkj jo}) positicvns o.f drivery loader, c|unner, arid 

t. Mik o, >rTirnaa r . 'rhr^' taa:', t wav- to bo a cnaaup tciat wi.tn mdivj .aaal T^- 

fa-' .at ( f ti Lt ■ t.c ] fnv l s ion a.crot ai w i Ma a :-i ta/ br:^ or ret i clo . 

'Vlv^ {dry aeloat:Lon -d:op wa:-:^ to ask vari.ovas mili.tary trairiinq 
a-jpa ! Mihan r a (yujMi^^ry, au tr^-no t l vc r -^^ind .^^uc!-!) to submit a list of critical 
ta::-.ka wioi ca aiioul l la,^ toa ted . Therjr dopartiriGnts Gu^ [ ted a total of 75 
(.a'^k-s. Bo-o^naa"} only a limitod ruami:)er of tasks ecu . bo used on the final 
tapt^, ^a\o liat iiad to he par.o:! down coiisidcrably . ?'1any tasks v;ere elim- 
inatoci m orrLar r.o balance the tost amonq skill levels and crew do^ it ior^^-^ . 
l-'or example, AO of the tasks received frorrs th.e departments were for skill 
levoi 3 and only S of these raska were on thio final tapo- , Mc' ^'omaining 
exotvaa taska were elimina La-d a irrraly by decidinq to limit ^ ho ^est to tasks 
asaociate^d witi^ th.e actual operation of the tank. Critical areas such as 
draq abuse, first aid, leadership^ and tactical decisions^ and complex 
tasks, sucri as s]'\-tchinq ;n area map and tasks that reouired excessively 
long telc^viaion running timeas , were eliminated. 

Thr first step in developing specific tost items for each task was 
"O list- all response components making up a task and decide wuether eroai 
ponse component was primarily cogni.tive^ perceptual, or motor. Fr^ach 

ooanitive or perceptual res^'-oiae component was then examined for criticality 

^ 
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The error- .(^tocticn r onse t:ypo was i]iclud(.:,. to i;est the examinee :•• ' 
kr^owledcjo of incorrect as .-el,] .. . cr:-rroct actions;. Manv incorrect actiuiis 
occur very inLrequently but caii bo very i:ericus; wher^i •'licy do occur. 
Pc.^r foniiinq the correct action in a hand:.-:-on t'j-Jt docn not necx: --^rMa r; i 1 y 
indicate awarorte^^s of dari :cr points, and exariiinc" need to be tested 
dircvjtly as to awareness 'f incorrect actions. j'owever; as Shriver 
points out, watchinq somoc-r:-' c^lse perform is very different from doi: .j 
It yourse], f. i-'ailinq to not.i .s: error rnav indi.vite ] ac]; of kn'.. .- t edcir; or 
it may iru:ti<:atL} inai^ility to lu*)!' i'.":* ^ *','rrGr \.]\ ^"'tiier. . 

ThL' overall crH:cd us i ns i -3 t.iia.t ' rror d. - ^ ' -'.ion s ;''.'U})tful re syv->:"i:-e 
ty[)e and more ti)ouq!U: and rescr(!.rcl'i are rieedi.-^d oidr^r :"o i l-.s a^^c^^: ^ tanc; as 
a useful t-rocedure. 

3. M otor man ipulation , This respons type was rather sr^-^cific to this 
Particular study and the response equipment being evaluated. In fact, on-.- 
..tronq selling point of tl'io response equipment v;as its provision for 
testing the motr- ■-nian i pu lat i on item.;. All test items under vhis [)articula:' 
respo: type i"^i:r taiini^d to wiv ^-e ehc examinee should {dace reticle on 

th</ t'MC'vision scre^si when simulating "fairing the main g is under v.^r-lo.,. 
conditions, Kowevc^r, analysis reveals that th:.is res- 'S"'- type is s's 
really a test of moL;_>r ability, but rat:M:r a test of a combin.ation of 
perceptual cisd cognitive abilities. The cognitive ulL-ni-aiit v;as knowing 
the correct lead aj^.d elevation for each target and the perceptual element 
was being ai^le to ^11 scrimi na t c the correct lead and elevation. There is 
no evide: t ■ i • :Le that tlie ability to manipulate a plastic reticle 
on a tele vis i'-v screen has any correlat'^in with the motor element involved 
in aiming an actual gun. On the plu-; side, triis response type is more of 
a recall item than a multiple-choice question and therefore should provide 
a more exaci me. 'l' of recall. On the r^'dmi:^ side:- is the requirement 
to learn a new re. onse qu.ckly (mianipula ting plastic reticles). Incorrect 
responses may be causr^d by lack of knowledge or perceptual ability, or 
merely by fa.ilure to master the new response of manipulating plastic 
reticles . 

This response type, like the error-detection res: ase type, needs 
much more thought and research prior to tan je as a us..fijl procedure. 

Some cominents on a few miscellaneous topics may also be useful: 

I. qr^ e of a time period to indicate err or. One item (3, Tables 1 
and G) -n the television tost used the pass.-ige of time as the cue for 
the exaiiiinee to note an err.^;r. That is, the actor waited only 5 seconds 
before continuing a procedure-, where the prescribed procedure in tP.e 
technical manual calls for a 120-180-3econd v;ait. Some cr.it.--cism has 
indicated that this time.-pa5:;age technique might confuse exar mees 
because Americans have been conditioned through exposure to mtoti 'i 
pictures and televisi )ii to accept any length time period shown oii a 
scre^Mi as the ^ap propriate time. Item '3 did seem to confuse many examinees. 
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2. Lo_n_^^ ' ^Sm^- Sevf^ral items on the trelovision tape had rclativeiv 
long runhi'H.j ,J.mos (approximately 3 minutes). Some critics claim that 
incli-idincr s^.r-h l^-.ncj items may be unv^isc because coverage of the tot:al 
subject mati-^.T is resr. :.jL-.'r] at best, given a 50-60 minute time limit 
for the test. Alijiouqh a lonq item may not necessarily confuse the 
examinees, it . .j noteworthy that 3 out of 4 long items retained in tlie 
test did prove very difficult for the television exarni jioes and all 4 
of th.e lonq items omitted from the study appeared to be confusincr 
;r.iriq pilt.;t ri.i,is. Another reason for omittinq long items is the 
i-fficulty iri getting an actor to i^e^rform a long sequence in letter- 
:.>ertr' rashion. Orre very long item on the tape ( evacuate injur e d 
c rewiVi.i n ) was never coinpletely satisfactory. The final ta^-e was 
acce: ;_ed l)^.'cause the director b'-came concerned with the safetv of the 
actor playing the role of injured cre^an. 

^' i^Jl- Unlike the human eye, television cannot capture 

both wide-annle view and good r- solution at the same time. For 
scon'-.; that require grod. resolution it is a good idea to zoom in on a 
sccM.:.^ and remain tliere. To attemj^t to show more than one closeup m 
any r^iic sequence tends to confuse the viev/er. 

Restr ic ted view . Ev-r:\ with a wide-ancle lens, television gives 
a- very restricted \^iew and care must be taken to provir' setting shotic. 
Precise judgments as t- the placement of controls are diffic.lt to 
make from a television pif-:ture- 

5; Poor__d_ep th perception . Much depth perception, is lost in a 
television picture- Items that depend apon judr:mGnt of depth should 
be omitted. 



Closeup and motion ^ Any kind of motion in a closeup shot is 
confusing. Necessary movements must be -.-ry slow and precise. However, 
it should be noted that slowness is per ^ived is an error by many people. 

More research needs to be accumulat :d before a more precise set of 
guidelines can be prod^- For television testing. Particularly needed 
is development of a mo.e adequate stimulus presen tat loi^. and response- 
recording device. Also needed are researchers well grounded in the 
capabilities and limitations of television and the use of television 
cameras, lighting, and editing equipment. Television testing offers 
much p-^tential but before this potential can be reached, much preparatory 
work r lains to be done . 
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EXAMPLES OF PAPER- AND-PENCIL ITEMS 



A . Mu ltiplo choi ce . 

12. You are the driver of an Mf^OAl tank. What response do you make to 
the followinq ground guide signal given at night with a flashlight? 



a. Move backward 

b* Start engine, 

c. Stop engine. 

6. Turn. left. 



B. Error Detection 




L 



0. You are the loader in an M60A1 tank. The tai^k commander gives the 
following fire cominand: 

/■' 

"GUNNER, BEEHI\/E TIME, TROOPS, ONE SIX HaNDRED" 



The tiring switch has l-een checked and the i 
six steps in order: 



he breecli is 



You do the 



il) Select, a BEEHIVE round. 

(2) Insert the round 2/3 of the way into the chamrjer, 

(3) Push the rour.^ into the chamber with the heel of 

(4) Clear the path of the recoil- 

(5) Turn the firing switch to FIRE. 

(6) Announce "UP." 

Did you do anything wrong? 
a. Step (2) is wrong. 

' Step (3) is wrong, 
c.'^^A step is missing. 



^ght hand- 



All of the steps are correct. 
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. Motor Maniif)ulatiori (Reti cle Manipulation) 



FOR THE NEXT FcIGr PROBLEMS ENGAGE ALL STATIONARY TARGETS Ay CENTER OF MASS 
AND ASSUME ALL | MOVI NG TARGETS ARE TRAVELING AT 15 MPH. 

y 

23. You are ^hc gunner on an MGOA.l tank. The tank cornmander gives the fir 
coiDinand : ■ 

"GL?-INER, SABOT, TANK" 

Which ryi the following sigiit pictures would you take up using the periscope 
reticle? 



I 
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APPENDIX C 



EVALUATION OF THE TELE5TRAT0R EQUIPMENT^ 

One of the reasons the television test was designed and produced 
was to evaluate the Telestrator equipment (also known as the Telestar 
equipment) . The novel component of this equipment is an electronic 
tablet which can be fitted over the .face of a television screen- The 
tablet will record the horizontal and vertical (XY) location when it 
is touched with an electronic contact point embedded in a stylus or 
similar indicator (such as a gansight reticlv.)- By the proper use of 
auxiliary recording equipment it is possible to record the place and 
time the screen is touched. Tne recording equipment includes a counter 
which keeps a running to> .il of the number of items, number of answers 
attempted, and number of correct answers - 

The complete system includes the electronic tablet, a programing 
unit, and several student units. The prograrrirrang unit is used by the te 
developer to place electronically on the television tape the XY 
coordinates for the correct answer for each test item and the time 
period during which the equipment will accept this answer. The student 
unit com.pares electronically the programed answer and the examinee's 
answer an 1 records the ^results. 

The student unit provides three types of feedback to the examinee 
for each test item. Immediate feedback is provided by a high-pitched 
tone and a small red light that comes on for a correct answer versus 
a low-pitched tone 'and no light for an incorrect answer. Slightly 
delayed feedback cgmes from a counter which shows new totals of items 
and correct answers' at the end of "t.he programed time period for 
ansv;ering each problem. 

As to whether the Telestrator equipment has any merit or not, it 
is necessary to exami.ne both: 

1- The equipment itself, as designed and produced, and 

2. The overall testing strategy which includes (a) Individual 

responding, (b) Television stimulus, (c) I/iimediate feedback^ » 

and (d) Time limit on each response. 

A. The Equipment ' 

As with most newly designed equipm,ent, the Telestrator contained 
many bugs and never worked properly. However,, it was possible to test 
some aspects of the equipment by using human graders to record right 
or wrong answers by the examinees according to the time and place the 
screen was touched- Several pi.Iot tests were run with the following 
results. 



^This summary oi Telestrator operation was submitted previously to the 
Training Support Division, TRADOC. 
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(1) Accuracy. There i.s a fundamental flaw in the Tolestrator 
design insofar cis precise respondinq i:^ concerned. The equipment v ?s 
claimed to he accurat^-^ to 1/4 inch. However, due to parallax the 
actual accuracy was r.\r>r:c on the order of 1 or 2 inches. This gross - 
nes:s effectively c 1 iminat*:!"! trie use of the reticle test items because 
with any reasonable 5'ize reticle no discrimination was possible for 
leads or rar-.;e-'- . The grossness also ulimiinated many test items in 
which the examinee v/as required to discriminate among several tank 
contr^ Is. ' The s};.icing betweiMi these controls as shown on the screei. 
was not gro-at enough to permit exact programing of the answers, and 
one answer box would overlap another. The parallax results from 
mounting the electronic tablet at some distance from the actual 
t'.'levision screen (due to curvature of the television screen the parallax 
increases as one approaches the edge of tb.e screen). 

In c.v.ier to continue the experiment and test the idea of responding 
to television, the electronic tablets were removed from the television 
monitors and the examinees were required to touch the face of the 
actual television screciis . This completely eliminated all parallax and 
permitted the use of reticle items and other precision responding. 

(2) Video presentation. The electronic tablet is constructed in 
such a manner Lhat it blocks a l-inch-wide area around the outer edge of 
the television screen. This is a serious limitation because it is 
necessary to use small-size monitor for such cioseup work and tiiis 
outer 1 inch covers a substairt^al parr of the aval lable " screen area. 



B. Te^t^inq Strategy 




Because of the device's failure to v/ork properly and the poor design 
of the electronic tablet, it was not possible to evaluate the testing 
strategy completely. However, by eliminating the parallax (removing 
the electronic tablets) and using riuman graders to record responses, 
it was possible to make a limited test of the strategy. 

(1) Responding to television. The examinees seemed to h.ave a little- 
trouble understanding the test items, and responded very precisely. 
Tljree types of test items were used; Multiple clioice. Error detection, 
and Reticle manipulation. 

No training other than instructions was required for learning to 
respond to the multiple-choice and error-^detection^ items* Approxim.ately 
10 minutes were required for training on the reticle-manipulation items. 

(2) Time to respond. Ten seconds were allowed for responding to 
each item. The time limit was generous and most examinees responded to 
most items in less than 5 seconds, 

(3) Perception of test. The examinees perceived the test as being 
"fair" and most actually preferred the television test over the hands-on 
test. 
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(4) Comparison with other ':'St-s. tolt;>visi.?:i tost was compared 
to parallel r^aper-and-ponci 1 and Ivinds-on tests. 

(a) Paper-and-pencil tost. Overall there war-i 1 i 1 1 le di f: f erence 
:jetw..-'m the noan scores on the television and the paper-and-p- .icxl. tests. 
Mow-v-r, on an i tem-by-- i terri basis there v;as conr; i d.eraM e ditLerenco for 
ii-.'-rn:,-.. Oa erro:!:- ■ "^ec tion ; th.e television score were mueh betcer; 
:;e'" <.:le rnanipulatior; the television scores were worse. 

(h') :dands-or' t-".:. There was a low [positive correlation between 
th-:. telev ision tes"^ -n^l tiie hands-on test . This correlation was less than 
desi.r.ild c . 

Tiie^ exr^erir: .it was too limited to perimit any conclusi.ons at this time 
wL:.]: reT''^- ■ r.^.' to the ^^-^ liabi 1 Lt y ami validit^/ of the above results. 

(5) t^e..'d}:)ack . because the evaminees were tested four at: a time and 
bccr:ra:a:,' tlie* 'j'elest rater equipment v;as not work' .^r it- v;as not ^'ossible 
to ; '^ro V i d(-"> i imne di ..i t c f e edba ck after e ach i te 

(e) rh/G fatigue. The television test and the responding mode 
r-' ''uired the examinees to stare continually at the television monitor. 
There were many cor^-.iaints of eyestrain and tliere is some evidence that 
the afternoon television examinees performed more poorly than the mcrn^nq 
te le v i si or. e xaminee s . 



C onclu s ion s a nyi R e c omme n d a t i o n s 

(1) The Telestra:or equipment as presently desiqned sliould be rejected 
because of the paralla;< problem. 

(2) Tlie televj.sion metliod appears to offer enough promise to warrant 
the testinq of other resijonse devices which do not have ehe parallax 
problem . 

(^) There are many unknowns in television testing and the overall 
testinq strateqy, and the research effort needs to be greatly expanded 
sue a as: 

(a) A more definiti\- comparison with hands-on tests. 

(b) Research on the "immediate feedback" idea. 



(c) iJsing alternative response devic^e.e 

(d) Comparison with slide-tape devi.ces. 

(e) Further research on eye fatigue. 
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